feat: Add Isolate kit - #291
Conversation
WalkthroughThe PR adds the Isolate reproduction kit. It includes Lamatic planning, Daytona sandbox certification, MCP and HTTP interfaces, a Next.js workbench, safety and evidence contracts, deployment configuration, documentation, and Bun tests. ChangesIsolate reproduction workflow
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
:robot_face: AgentKit Structural ValidationNew Contributions Detected
Check Results
🎉 All checks passed! This contribution follows the AgentKit structure. |
|
Failure recorded at 2026-07-24T10:48:07Z UTC. If this PR is not fixed within 4 weeks it will be automatically closed. |
There was a problem hiding this comment.
Actionable comments posted: 33
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kits/isolate/agent.md`:
- Around line 19-22: Update the supported-scope statement in the capability
document to include Bun alongside Node.js and TypeScript, while preserving the
existing repository and modification restrictions.
In `@kits/isolate/apps/lib/concurrency.ts`:
- Around line 1-14: Extend acquireInvestigationSlot beyond the process-local
activeInvestigations counter by backing it with a shared limiter such as
Redis/Upstash or a provider-side quota, while retaining the local gate as a
fallback when the shared limiter is unavailable. Document in the kit README that
the local maximumConcurrentInvestigations limit is instance-scoped and does not
cap concurrency across autoscaled Vercel instances.
In `@kits/isolate/apps/lib/investigate.ts`:
- Around line 118-140: The model-call retry logic retries every failure
immediately and unconditionally. In
kits/isolate/apps/lib/investigate.ts:118-140, extract a shared retry helper that
classifies errors, stops immediately for deterministic failures, applies backoff
for transient or rate-limit failures, and respects the remaining deadline;
update requestPlan to use it. In kits/isolate/apps/lib/investigate.ts:216-225,
replace the unconditional requestReport retry with the same helper, preserving
the existing requestReport behavior while applying identical error handling.
- Around line 226-240: Restrict issueRequiresInteractiveEvidence in the
investigation flow to explicit interaction signals such as keypress, keyboard,
TUI, or pressing a control, removing generic matches for save, saving, editor,
click, type, and typing. Keep the existing report.limitations evidence-gap check
and inconclusive outcome handling unchanged.
- Around line 18-29: Cap the repository snapshot output before constructing the
planner context, using the same 45,000-character bound as the evidence/report
payload. Update the relevant investigate flow to derive and reuse
repositoryContext from the bounded snapshot, replacing direct
snapshot.observation.stdout usage at the claim-derivation, planner, and
subsequent context call sites.
- Around line 327-330: Update the cleanup flow surrounding runtime.delete so a
deletion failure preserves the original investigation error when the main
operation already failed, while still failing closed when deletion cannot be
confirmed. Restructure the success paths to delete before returning, or
otherwise combine both failures without replacing the original cause, using the
existing investigate flow and runtime.delete call.
In `@kits/isolate/apps/lib/lamatic-planner.ts`:
- Around line 72-81: Update the response-handling flow around response.json() to
parse defensively, catching malformed or empty upstream bodies instead of
allowing a raw parsing error to escape. In the planner request logic, ensure
response.ok and the parsed error message determine the thrown error, falling
back to “Lamatic could not produce a probe plan.” when the body cannot be parsed
or has no usable message; preserve successful execution handling via
executeWorkflow.
- Around line 55-71: Update executeWorkflow to invoke the sanctioned lamatic SDK
flow-call API instead of manually constructing the GraphQL request,
authentication, and project headers. Preserve the existing input,
policyFeedback, endpoint, and per-request timeout behavior; if the SDK cannot
accept the timeout signal, document that limitation with a brief comment and
dependency rationale.
In `@kits/isolate/apps/lib/runtime/claim.ts`:
- Around line 50-71: Restrict the tui_unsaved_exit classification in the claim
logic so repository-wide context cannot independently satisfy the key signals.
Require namesQuitShortcut to match the issue text, while allowing
namesTerminalSurface to use contextualText only as corroboration; preserve the
existing namesExit and namesUnsavedState checks.
In `@kits/isolate/apps/lib/runtime/daytona.ts`:
- Around line 668-676: Update the cleanup in the installation flow’s finally
block to catch and log failures from updateNetworkPolicy when re-blocking the
network, while preserving the original installation error. Track whether the
main body succeeded, and only rethrow the cleanup failure when no body error
occurred, matching the behavior used by runProbe.
- Around line 394-417: Update runProbe in
kits/isolate/apps/lib/runtime/daytona.ts at lines 394-417 to capture the body
error, log and swallow cleanup failures, rethrowing the body error first and
surfacing cleanup failure only when the body succeeded. At
kits/isolate/apps/lib/runtime/daytona.ts lines 298-324, attach logging to
deleteSandbox cleanup so “Repository cloning failed.” remains the propagated
error; at lines 668-676, log and swallow re-block updateNetworkPolicy failures
so “Deterministic dependency installation failed.” survives. In
kits/isolate/apps/lib/runtime/mcp.ts lines 198-219, log and swallow
runtime.delete failures in the inner finally so completed verdicts are returned.
- Around line 550-565: Make the TUI readiness polling around the
alternate-screen marker an explicit success condition before sending any
keystrokes. Track whether output contains "\u001b[?1049h"; if the 40-attempt
loop completes without detecting it, fail or throw through the existing
certification error path and do not execute the subsequent ESC, edit, save, or
quit sends. Preserve the current interaction sequence only when readiness is
confirmed.
- Around line 298-324: Update the catch block in the sandbox setup flow so the
original cloning or network-policy error remains the thrown error when
deleteSandbox cleanup fails. Catch cleanup failures from deleteSandbox, preserve
the original error, and attach the cleanup failure as its cause or otherwise
retain it without replacing the primary error.
- Around line 378-386: Update the collection flow around execute and the
observation parse to validate collected.exitCode before calling JSON.parse,
matching the guarded behavior in runTuiUnsavedExitProbe. On nonzero exit, fail
closed with a diagnosable error identifying the collector probe and relevant
stderr; only parse and validate the JSON result when execution succeeds.
In `@kits/isolate/apps/lib/runtime/github.ts`:
- Around line 49-58: Update the request options in the issue-reading method
containing the GitHub API call so the caller-provided options.signal is composed
with a separate 10-second AbortSignal.timeout(10_000), rather than selected via
nullish coalescing. Preserve both cancellation sources: caller deadlines must
still abort the request, and the GitHub fetch must always enforce its own
10-second timeout.
In `@kits/isolate/apps/lib/runtime/investigation-report.ts`:
- Around line 3-4: Update investigationReportSchema so model output cannot
define the final reproduction classification: remove the outcome field and
inject the deterministic certification result when constructing the report, or
replace it with the canonical runtime-owned certification enum. Ensure the
report uses the certified result rather than the model-specific
likely_reproduced value.
In `@kits/isolate/apps/lib/runtime/mcp.ts`:
- Around line 232-251: Harden the authorization gate in the MCP request handler
by parsing the Authorization header into scheme and token, accepting the scheme
case-insensitively, and comparing the token to secret with
crypto.timingSafeEqual only after confirming equal-length buffers. Preserve the
existing 401 response for missing, malformed, or mismatched credentials.
In `@kits/isolate/apps/lib/runtime/policy.ts`:
- Line 8: Update the environment-dump regex in the policy tripwire to match env,
printenv, or set only when they occur in executable command position, following
the existing sudo rule’s anchoring approach. Preserve rejection of direct
environment-dump commands while allowing those words in script names, arguments,
paths, and subcommands.
In `@kits/isolate/apps/package.json`:
- Around line 13-14: Update the `@modelcontextprotocol/server` dependency in the
package manifest to use an exact pinned beta version instead of the caret range,
and apply the same deterministic versioning to the matching
`@modelcontextprotocol/client` dependency. Preserve compatibility between the two
MCP packages and ensure neither can resolve to an unintended pre-release update.
- Line 26: Update compilerOptions.types in kits/isolate/apps/tsconfig.json to
include node, react, and react-dom alongside bun so the app can resolve the
expected type packages under TypeScript 6.0.
In `@kits/isolate/apps/tests/claim.test.ts`:
- Around line 9-36: Add focused assertions in the “issue evidence contract”
tests that exercise extractIssueEvidenceAssertion with an Observed stderr
backtick value and with an unquoted plain value, verifying the expected
stderr_contains and stdout_contains results respectively. Keep the existing
rejection cases unchanged.
In `@kits/isolate/apps/tests/concurrency.test.ts`:
- Around line 11-21: Extend the test named “bounds expensive work per
application instance” to invoke the same release closure twice and verify that
the second invocation does not free another slot. Confirm the slot remains
unavailable until the other acquired slot is released, then preserve the
existing assertion that a slot becomes available after a legitimate release.
In `@kits/isolate/apps/tests/evidence.test.ts`:
- Around line 33-85: Add tests in the certifyEvidence suite covering
candidateRuns with one run and with three runs, asserting neither is certified
as reproduced and that the gate repeatCount reflects the supplied arity. Keep
the existing two-run certification test unchanged so the exact repeatCount === 2
boundary is pinned.
In `@kits/isolate/apps/tests/investigate.test.ts`:
- Line 460: Rename the tests at the indicated locations to clearly state that
the outcome is downgraded to “inconclusive” when the limited exploratory probe
is not reproduced and when the static launch likely indicates an interactive
reproduction. Keep the test behavior and assertions unchanged.
- Around line 174-195: Add a test case for an explicit issue ref through
investigateIssue, configuring the issue input with a non-default ref and
asserting the ref is included in the runtime.create inputs and passed to the
planner. Preserve the existing default-branch test and verify the investigation
still uses the requested ref.
In `@kits/isolate/apps/tests/investigation-report.test.ts`:
- Around line 5-21: Extend the parseInvestigationReport tests with rejection
cases for malformed JSON, an unsupported outcome value, and a report missing
markdown. Assert each input throws or is otherwise rejected, while preserving
the existing valid bounded-report test.
In `@kits/isolate/apps/tests/investigation-request.test.ts`:
- Around line 20-36: Add tests covering the rejection behavior of
investigationRequest for a pull request URL, a non-GitHub issues URL, and a
malformed URL. Assert the function’s actual failure contract—return null or an
unsuccessful safeParse result rather than throwing, as implemented—while
preserving the existing valid-URL cases.
In `@kits/isolate/apps/tests/lamatic-planner.test.ts`:
- Around line 55-76: Add failure-path tests for requestLamaticReport covering
non-ok HTTP responses, status values other than “success,” and passthrough of
errors[0].message. Add assertions that the request sends the expected
Authorization and x-project-id headers, preserving the contract used by
investigate.ts retries.
In `@kits/isolate/apps/tests/mcp.test.ts`:
- Around line 41-57: Expand the authorization tests around handleMcp to cover
both a presented but incorrect bearer token and a defined authorization header
when the configured secret is undefined. Assert that each request is rejected
with HTTP 401, preserving the existing missing-header case and verifying exact
token matching plus fail-closed behavior.
In `@kits/isolate/apps/tests/plan.test.ts`:
- Around line 63-178: Refactor the single test around assertCertificationCommand
into table-driven rejection cases executed with test.each, giving each command
vector a descriptive name and expected output where needed. Keep the existing
accepted-command assertions separate, and preserve every current rejection
vector so failures are reported independently with the specific escape hatch
identified.
In `@kits/isolate/apps/tsconfig.json`:
- Around line 2-15: Enable noUncheckedIndexedAccess in the compilerOptions of
tsconfig.json, then resolve the resulting type errors across the affected
token-parsing code, including the matches[0] destructuring in claim.ts, by
explicitly handling potentially undefined indexed values while preserving
existing command-safety behavior.
- Around line 16-18: Update the compilerOptions.types configuration in
tsconfig.json so the Next app does not globally include Bun types; remove the
explicit "bun" entry or replace it with the appropriate Next-specific types
configuration while preserving the rest of the TypeScript settings.
In `@kits/isolate/lamatic.config.ts`:
- Line 24: Update the docs entry in the Isolate kit configuration to remove the
supervisor-agent URL; either point it to an existing Isolate-specific landing or
README URL, or remove the docs property until such documentation exists.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 960caa46-f778-499f-a2d4-5ce4be8e25ae
⛔ Files ignored due to path filters (2)
kits/isolate/apps/bun.lockis excluded by!**/*.lockkits/isolate/assets/isolate-evidence.jpgis excluded by!**/*.jpg
📒 Files selected for processing (55)
kits/isolate/.env.examplekits/isolate/.gitignorekits/isolate/README.mdkits/isolate/agent.mdkits/isolate/apps/.env.examplekits/isolate/apps/.gitignorekits/isolate/apps/DESIGN.mdkits/isolate/apps/PRODUCT.mdkits/isolate/apps/app/api/investigate/route.tskits/isolate/apps/app/api/mcp/route.tskits/isolate/apps/app/globals.csskits/isolate/apps/app/layout.tsxkits/isolate/apps/app/page.tsxkits/isolate/apps/components/investigation-workbench.tsxkits/isolate/apps/lib/concurrency.tskits/isolate/apps/lib/deadline.tskits/isolate/apps/lib/http-errors.tskits/isolate/apps/lib/investigate.tskits/isolate/apps/lib/investigation-request.tskits/isolate/apps/lib/lamatic-planner.tskits/isolate/apps/lib/runtime/certification.tskits/isolate/apps/lib/runtime/claim.tskits/isolate/apps/lib/runtime/daytona.tskits/isolate/apps/lib/runtime/evidence.tskits/isolate/apps/lib/runtime/github.tskits/isolate/apps/lib/runtime/investigation-report.tskits/isolate/apps/lib/runtime/mcp.tskits/isolate/apps/lib/runtime/plan.tskits/isolate/apps/lib/runtime/policy.tskits/isolate/apps/lib/runtime/probe.tskits/isolate/apps/next-env.d.tskits/isolate/apps/next.config.tskits/isolate/apps/package.jsonkits/isolate/apps/tests/certification.test.tskits/isolate/apps/tests/claim.test.tskits/isolate/apps/tests/concurrency.test.tskits/isolate/apps/tests/daytona.test.tskits/isolate/apps/tests/deadline.test.tskits/isolate/apps/tests/evidence.test.tskits/isolate/apps/tests/github.test.tskits/isolate/apps/tests/http-errors.test.tskits/isolate/apps/tests/investigate.test.tskits/isolate/apps/tests/investigation-report.test.tskits/isolate/apps/tests/investigation-request.test.tskits/isolate/apps/tests/lamatic-planner.test.tskits/isolate/apps/tests/mcp.test.tskits/isolate/apps/tests/plan.test.tskits/isolate/apps/tests/probe.test.tskits/isolate/apps/tsconfig.jsonkits/isolate/constitutions/default.mdkits/isolate/flows/isolate-reproduction.tskits/isolate/lamatic.config.tskits/isolate/model-configs/isolate-reproduction-model.tskits/isolate/prompts/isolate-reproduction-system.mdkits/isolate/prompts/isolate-reproduction-user.md
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
kits/isolate/apps/lib/runtime/daytona.ts (1)
344-436: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winBiome's
noUnsafeFinallyis still tripping on lines 433 and 711 — the restructure didn't actually clear it.Both blocks now correctly preserve the primary error/log the cleanup failure per the earlier review's intent, but Biome flags any
throwinside afinally, conditional or not (confirmed: its rule engine doesn't reason about the surroundingif). The static analysis hints for this diff still show[error] 433-433and[error] 711-711for "Unsafe usage of 'throw'", so this will keep failing lint even though the runtime semantics are sound now.Move the "throw only if the body succeeded" decision outside the
finallyblock entirely to satisfy both the linter and the original request.🧹 Move the conditional throw out of `finally`
- } catch (error) { - probeError = error; - throw error; - } finally { + } catch (error) { + probeError = error; + } finally { let cleanupError: unknown; let cleaned = false; for (let attempt = 0; attempt < 2; attempt += 1) { ... } - if (!cleaned) { - if (probeError) { - console.error("Isolate probe cleanup failed after probe failure", cleanupError); - } else { - throw cleanupError; - } - } + if (!cleaned && probeError) { + console.error("Isolate probe cleanup failed after probe failure", cleanupError); + } + this.pendingProbeCleanupError = !cleaned && !probeError ? cleanupError : undefined; } + if (probeError) throw probeError; + if (this.pendingProbeCleanupError) throw this.pendingProbeCleanupError;Apply the analogous restructuring to
resetWorkspace's try/catch/finally around line 675-715.Also applies to: 675-715
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kits/isolate/apps/lib/runtime/daytona.ts` around lines 344 - 436, Restructure the probe cleanup flow so the finally block performs cleanup and records cleanupError but contains no throw; after the try/finally completes, conditionally throw cleanupError only when no probeError occurred, while preserving primary probe errors and cleanup logging. Apply the same pattern to resetWorkspace’s try/catch/finally cleanup flow, ensuring both paths satisfy noUnsafeFinally without changing error precedence.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kits/isolate/apps/lib/investigate.ts`:
- Around line 63-82: Update assertRequiredLocalServiceStarted to detect
arbitrary repository service-script names instead of only the literal service,
server, and api keys, so local-service commands require the corresponding
package-manager run command. Preserve the existing local-URL and rejection
behavior, and escape the captured serviceScript before interpolating it into new
RegExp because repositoryContext is untrusted.
In `@kits/isolate/apps/lib/runtime/github.ts`:
- Around line 57-60: Update the request timeout handling around AbortSignal.any
to avoid relying on the affected runtime behavior: retain a strong reference to
the composed timeout signal for the request lifetime, or use a runtime-supported
equivalent that cannot be garbage-collected before firing. Preserve
options.signal composition and the 10-second timeout, and verify the deployed
Node.js runtime includes the required GC fix.
In `@kits/isolate/apps/package.json`:
- Around line 15-17: Update the dependency declarations in package.json to use
compatible Next.js 14–15 and React 18 releases, including matching react-dom and
both `@types/react` packages. Then regenerate the lockfile so it reflects the
selected versions and resolved dependency graph.
In `@kits/isolate/prompts/isolate-reproduction-system.md`:
- Line 8: Update the TUI unsaved-exit evidence contract to replace the
unsupported “bun --cwd” wording with the allowed structured workspace form “bun
run --cwd <relative-package-directory> <script>”. Preserve the existing
requirements for working-directory-relative artifact paths and all runtime-owned
fixture, input, assertion, repeat, and cleanup behavior.
---
Outside diff comments:
In `@kits/isolate/apps/lib/runtime/daytona.ts`:
- Around line 344-436: Restructure the probe cleanup flow so the finally block
performs cleanup and records cleanupError but contains no throw; after the
try/finally completes, conditionally throw cleanupError only when no probeError
occurred, while preserving primary probe errors and cleanup logging. Apply the
same pattern to resetWorkspace’s try/catch/finally cleanup flow, ensuring both
paths satisfy noUnsafeFinally without changing error precedence.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 39f9c593-c9b1-4fa4-9812-1fbd0bec986d
📒 Files selected for processing (24)
kits/isolate/README.mdkits/isolate/agent.mdkits/isolate/apps/lib/investigate.tskits/isolate/apps/lib/investigation-request.tskits/isolate/apps/lib/lamatic-planner.tskits/isolate/apps/lib/runtime/claim.tskits/isolate/apps/lib/runtime/daytona.tskits/isolate/apps/lib/runtime/evidence.tskits/isolate/apps/lib/runtime/github.tskits/isolate/apps/lib/runtime/mcp.tskits/isolate/apps/lib/runtime/policy.tskits/isolate/apps/package.jsonkits/isolate/apps/tests/claim.test.tskits/isolate/apps/tests/concurrency.test.tskits/isolate/apps/tests/daytona.test.tskits/isolate/apps/tests/evidence.test.tskits/isolate/apps/tests/investigate.test.tskits/isolate/apps/tests/investigation-report.test.tskits/isolate/apps/tests/investigation-request.test.tskits/isolate/apps/tests/lamatic-planner.test.tskits/isolate/apps/tests/mcp.test.tskits/isolate/apps/tsconfig.jsonkits/isolate/lamatic.config.tskits/isolate/prompts/isolate-reproduction-system.md
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
kits/isolate/apps/lib/investigate.ts (1)
161-164: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winMission update: the "bun --cwd" briefing never got the memo.
This runtime-injected
evidenceGuidancestring still tells the planner "whencommandusesbun --cwd, adjust the path" — the exact wording already corrected tobun run --cwdinkits/isolate/prompts/isolate-reproduction-system.md. Left as-is, a TUI-mode plan following this instruction literally can emit the unsupportedbun --cwdrunner form, which the command policy rejects, costing a plan-repair retry for the wholetui_unsaved_exitevidence path.🕵️ Sync the wording with the fixed prompt file
-Resolve every build-artifact path from the effective working directory of command; when command uses bun --cwd, adjust the path for that directory. +Resolve every build-artifact path from the effective working directory of command; when command uses bun run --cwd, adjust the path for that directory.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kits/isolate/apps/lib/investigate.ts` around lines 161 - 164, Update the `evidenceGuidance` text for `assertion?.kind === "tui_unsaved_exit"` to refer to the supported `bun run --cwd` form instead of `bun --cwd`, while preserving the existing path-adjustment instruction and all other guidance.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@kits/isolate/apps/lib/investigate.ts`:
- Around line 161-164: Update the `evidenceGuidance` text for `assertion?.kind
=== "tui_unsaved_exit"` to refer to the supported `bun run --cwd` form instead
of `bun --cwd`, while preserving the existing path-adjustment instruction and
all other guidance.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 783ba816-1b42-45f9-8a7e-4ace6c7baa69
⛔ Files ignored due to path filters (1)
kits/isolate/apps/bun.lockis excluded by!**/*.lock
📒 Files selected for processing (13)
kits/isolate/apps/lib/investigate.tskits/isolate/apps/lib/runtime/daytona.tskits/isolate/apps/lib/runtime/github.tskits/isolate/apps/lib/runtime/investigation-report.tskits/isolate/apps/next-env.d.tskits/isolate/apps/package.jsonkits/isolate/apps/styles.d.tskits/isolate/apps/tests/investigate.test.tskits/isolate/apps/tests/investigation-report.test.tskits/isolate/apps/tests/lamatic-planner.test.tskits/isolate/apps/tests/plan.test.tskits/isolate/apps/tsconfig.jsonkits/isolate/prompts/isolate-reproduction-system.md
|
@akshatvirmani Done: the PR is ready for final review. All CodeRabbit findings are addressed and resolved, CodeRabbit approved the exact current head, repository validation and the full local test/typecheck/build gate pass, and the production demo is deployed and Ready. |
|
Hi @Dhruv2mars! 👋 Before this PR can be reviewed by maintainers, please resolve all comments and requested changes from the CodeRabbit automated review. Steps to follow:
This helps keep the review process efficient for everyone. Thank you! 🙏 |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
kits/isolate/apps/lib/investigate.ts (1)
280-289: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winKeep the Agent hypothesis field consistent with the remaining investigation paths.
This exploratory evidence review assigns
report.summarytohypothesis, while certified and TUI paths return the planner hypothesis and the UI labelsresult.hypothesisas the preliminary Lamatic hypothesis. Use the planner hypothesis if this field must continue representing the agent’s planner hypothesis; otherwise adjust the field name/contract and update the UI to prevent calling the post-runtime summary an agent hypothesis.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kits/isolate/apps/lib/investigate.ts` around lines 280 - 289, Update the exploratory evidence-review return object in the investigate flow to use the planner’s hypothesis for the `hypothesis` field, matching the certified and TUI paths, instead of `report.summary`. Preserve the existing report summary in its appropriate report field, and keep the `result.hypothesis` UI label consistent with the value’s planner-hypothesis contract.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@kits/isolate/apps/lib/investigate.ts`:
- Around line 280-289: Update the exploratory evidence-review return object in
the investigate flow to use the planner’s hypothesis for the `hypothesis` field,
matching the certified and TUI paths, instead of `report.summary`. Preserve the
existing report summary in its appropriate report field, and keep the
`result.hypothesis` UI label consistent with the value’s planner-hypothesis
contract.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 3aa13f47-e669-4460-ba34-8fcf3a224640
📒 Files selected for processing (2)
kits/isolate/apps/lib/investigate.tskits/isolate/apps/tests/investigate.test.ts
Isolate
Live demo · Evaluation issue · Kit documentation
Problem
Open-source maintainers regularly receive issues that describe a symptom but omit the exact environment and commands needed to reproduce it. An AI agent can propose a plausible explanation, but its confidence is not evidence.
Isolate turns a public GitHub issue into a repeatable, machine-verifiable reproduction report inside a disposable sandbox.
What this kit adds
Trust boundary
The model investigates. The runtime verifies.
Lamatic forms a hypothesis and proposes a candidate probe plus a nearby negative control. It cannot mark its own work reproduced. The runtime derives one exact assertion from the issue and certifies
reproducedonly when:If that gate does not pass, Isolate returns
not_reproduced_under_tested_conditionsorblocked.Architecture
flowchart LR A[Public GitHub issue] --> B[Deterministic intake] B --> C[Disposable Daytona sandbox] C --> D[Lamatic probe planner] D --> E[Runtime command policy] E --> F[Candidate twice plus control] F --> G[Deterministic evidence gate] G --> H[Markdown and JSON report]Reviewer path
The fixture describes the symptom and observed output without supplying the reproduction command. Isolate must inspect the repository and discover the correct repository-owned CLI invocation.
Verification
Scope
The initial release supports public Node.js, TypeScript, Bun, and terminal/CLI issues. Terminal-output certification uses an exact issue-derived stdout/stderr signature. For unsaved-exit TUI issues, the runtime drives a real PTY, verifies unchanged file state twice, and rejects the claim after a save control. Other issue classes remain blocked without a runtime-owned evidence adapter. Isolate does not edit repository source files, generate fixes, mount repository credentials, push branches, open pull requests, or publish packages.
Challenge checklist
kits/isolateagentkit-challengelabel appliedStatus
Ready for final review. Implementation, submission materials, live demo, and reviewer walkthrough are complete.
Files Added
Configuration and documentation:
kits/isolate/.env.examplekits/isolate/.gitignorekits/isolate/README.mdkits/isolate/agent.mdkits/isolate/constitutions/default.md("Isolate Constitution")kits/isolate/lamatic.config.tskits/isolate/model-configs/isolate-reproduction-model.tskits/isolate/prompts/isolate-reproduction-system.mdkits/isolate/prompts/isolate-reproduction-user.mdLamatic flow:
kits/isolate/flows/isolate-reproduction.ts(Lamatic flow definition)Next.js application core:
kits/isolate/apps/.env.examplekits/isolate/apps/.gitignorekits/isolate/apps/DESIGN.mdkits/isolate/apps/PRODUCT.mdkits/isolate/apps/styles.d.tskits/isolate/apps/next-env.d.tskits/isolate/apps/next.config.tskits/isolate/apps/package.jsonkits/isolate/apps/tsconfig.jsonkits/isolate/apps/app/globals.csskits/isolate/apps/app/layout.tsxkits/isolate/apps/app/page.tsxAPI routes:
kits/isolate/apps/app/api/investigate/route.tskits/isolate/apps/app/api/mcp/route.tsFrontend components:
kits/isolate/apps/components/investigation-workbench.tsxRuntime and service libraries:
kits/isolate/apps/lib/concurrency.tskits/isolate/apps/lib/deadline.tskits/isolate/apps/lib/http-errors.tskits/isolate/apps/lib/investigate.tskits/isolate/apps/lib/investigation-request.tskits/isolate/apps/lib/lamatic-planner.tskits/isolate/apps/lib/runtime/certification.tskits/isolate/apps/lib/runtime/claim.tskits/isolate/apps/lib/runtime/daytona.tskits/isolate/apps/lib/runtime/evidence.tskits/isolate/apps/lib/runtime/github.tskits/isolate/apps/lib/runtime/investigation-report.tskits/isolate/apps/lib/runtime/mcp.tskits/isolate/apps/lib/runtime/plan.tskits/isolate/apps/lib/runtime/policy.tskits/isolate/apps/lib/runtime/probe.tsTest suite:
kits/isolate/apps/tests/certification.test.tskits/isolate/apps/tests/claim.test.tskits/isolate/apps/tests/concurrency.test.tskits/isolate/apps/tests/daytona.test.tskits/isolate/apps/tests/deadline.test.tskits/isolate/apps/tests/evidence.test.tskits/isolate/apps/tests/github.test.tskits/isolate/apps/tests/http-errors.test.tskits/isolate/apps/tests/investigate.test.tskits/isolate/apps/tests/investigation-report.test.tskits/isolate/apps/tests/investigation-request.test.tskits/isolate/apps/tests/lamatic-planner.test.tskits/isolate/apps/tests/mcp.test.tskits/isolate/apps/tests/plan.test.tskits/isolate/apps/tests/probe.test.tsAssets:
kits/isolate/assets/isolate-evidence.jpgLamatic Flow (
kits/isolate/flows/isolate-reproduction.ts)Node types introduced:
triggerNode_1(type:triggerNode, "API Request"): Receivesadvance_schemacontainingissue,repositoryContext,ref, andpolicyFeedbackfields from incoming API request.LLMNode_887(type:dynamicNode, "Generate Text"): Executes system and user prompts (@prompts/isolate-reproduction-system.md,@prompts/isolate-reproduction-user.md) using generative model configuration from@model-configs/isolate-reproduction-model.ts(Gemini 3.1 Flash Lite).responseNode_triggerNode_1(type:responseNode, "API Response"): Maps LLM output to JSON response by embeddingLLMNode_887.output.generatedResponseinto aplanfield withapplication/jsoncontent type.Data flow:
triggerNode_1forwards API request toLLMNode_887via default edge.LLMNode_887processes issue context and produces reproduction plan via LLM inference.responseNode_triggerNode_1receives LLM output and returns structured plan response via default edge.triggerNode_1toresponseNode_triggerNode_1for response routing.High-level behavior:
The flow receives a GitHub issue, repository snapshot, optional commit reference, and policy feedback. The LLM node interprets the issue as untrusted input and produces a structured reproduction plan that specifies either terminal-based candidate/control commands or TUI-based unsaved-exit probe instructions. The runtime then executes this plan in an isolated Daytona sandbox, certifies candidate and control probe evidence against the issue's observed assertion, and reports pass/fail outcome based on the evidence gate criteria (two candidate passes and negative control rejection).